Skip to content

[9.0] Fix JDK matrix pipeline after configurable it split (backport #17461) #17514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 8, 2025

Conversation

mergify[bot]
Copy link

@mergify mergify bot commented Apr 8, 2025

Release notes

[rn:skip]

What does this PR do?

PR #17219 introduced configurable split quantities for IT tests, which resulted in broken JDK matrix pipelines (e.g. as seen via the elastic internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.

)

This PR fixes the above problem.

Why is it important/What is the impact to the user?

Restores the functionality of the JDK (Windows/Linux) matrix pipelines.

How to test this PR locally

Linux

BUILDKITE_PIPELINE_NAME="Logstash Linux JDK matrix pipeline" MATRIX_OSES="ubuntu-22.04" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

produces:

Output from generate-steps for Linux
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
steps:
  - group: ubuntu-22.04/adoptiumjdk_21
    key: ubuntu-22_04_adoptiumjdk_21
    steps:
      - label: Initialize annotation
        key: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        command: |-
          buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 "### Group: ubuntu-22.04 / adoptiumjdk_21
          | **Status** | **Test** |
          | --- | ----|
          "
      - label: Java Unit Test
        key: ubuntu-22_04_adoptiumjdk_21-java-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport ENABLE_SONARQUBE=\"false\"\nci/unit_tests.sh java\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Java Unit Test |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Java Unit Test |\n\"\nfi\n      "
      - label: Ruby Unit Test
        key: ubuntu-22_04_adoptiumjdk_21-ruby-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/unit_tests.sh ruby\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Ruby Unit Test |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Ruby Unit Test |\n\"\nfi\n      "
      - label: Integration Tests - 1/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-1-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 0 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 1/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 1/3 |\n\"\nfi\n      "
      - label: Integration Tests - 2/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-2-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 1 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 2/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 2/3 |\n\"\nfi\n      "
      - label: Integration Tests - 3/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-3-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 2 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 3/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 3/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 1/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-1-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 0 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 1/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 1/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 2/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-2-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 1 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 2/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 2/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 3/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-3-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 2 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 3/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 3/3 |\n\"\nfi\n      "
      - label: x-pack unit tests
        key: ubuntu-22_04_adoptiumjdk_21-x-pack-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/unit_tests.sh\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack unit tests |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack unit tests |\n\"\nfi\n      "
      - label: x-pack integration
        key: ubuntu-22_04_adoptiumjdk_21-x-pack-integration
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/integration_tests.sh\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack integration |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack integration |\n\"\nfi\n      "

Windows

BUILDKITE_PIPELINE_NAME="Logstash Windows JDK matrix pipeline" MATRIX_OSES="windows-2025" MATRIX_JDKS="adoptiumjdk_21" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

results in the following output:

# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
steps:
  - group: windows-2025/adoptiumjdk_21
    key: windows-2025_adoptiumjdk_21
    steps:
      - label: Initialize annotation
        key: windows-2025-adoptiumjdk_21-initialize-annotation
        command: |-
          buildkite-agent annotate --style=info --context=windows-2025_adoptiumjdk_21 "### Group: windows-2025 / adoptiumjdk_21
          | **Status** | **Test** |
          | --- | ----|
          "
      - label: Java Unit Test
        key: windows-2025_adoptiumjdk_21-java-unit-test
        depends_on: windows-2025-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-windows-2025
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        artifact_paths:
          - build_reports.zip
        retry:
          automatic:
            - limit: 3
        command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Java Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 java\" -Annotate\n        "
      - label: Ruby Unit Test
        key: windows-2025_adoptiumjdk_21-ruby-unit-test
        depends_on: windows-2025-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-windows-2025
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        artifact_paths:
          - build_reports.zip
        retry:
          automatic:
            - limit: 3
        command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Ruby Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 ruby\" -Annotate\n        "

Related issues

PR #17219


This is an automatic backport of pull request #17461 done by [Mergify](https://mergify.com).

PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)
@mergify mergify bot added the backport label Apr 8, 2025
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

cc @dliappis

Copy link
Contributor

@dliappis dliappis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean backport LGTM

@dliappis dliappis merged commit fcf313a into 9.0 Apr 8, 2025
7 checks passed
@dliappis dliappis deleted the mergify/bp/9.0/pr-17461 branch April 8, 2025 14:02
@dliappis dliappis added the ci label Apr 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants